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ABSTRACT 

After establishing a theoretical depiction of the 
domain of literature learning, a study developed test packages which 
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differences according to the genres; (3) the relationship between 
literary and non-literary texts, both with respect to practice and 
with respect to aesthetic judgment; and (4) the concern about the 
scoring of open-ended responses, and whether the quality of writing 
could be distinguished from the quality of the understanding of the 
text. Tests were administered to a sample of tenth, eleventh, and 
twelfth graders in New York State. Results tended to validate the 
model that knowledge, practice, and preference are related but not 
highly interrelated aspects of the construct of literature learning, 
and that a comprehensive measure of student performance should 
address each of these three areas. It appears that within the 
knowledge domain, textual knowledge and knowledge of critical terms 
are distinct, particularly in their relationship to the practice of 
reading and responding. Within the domain of practice, more than one 
passage is needed to get some estimate of a stude.it' s performance 
across text-types. It seems to make little difference whether 
open-ended or multiple choice questions are used, but open-ended 
questions probably present somewhat more of a challenge to students 
than do multiple choice questions. (One table of data and one figure 
are included; sample measures, 20 references, and 28 tables of data 
are attached.) (PRA) 
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COMPARISON OF MEASURES OF THE DOMAIN OF LEARNING IN 

LITERATURE 



Alan C. Purves 

Hongru Li 
Margaret Shirk 
University at Albany 
State University of New York 

One of the major concerns about literature education in the United States of America is that 
concerning its goals and objectives, particularly as these are reflected in its assessment. In raising his first 
concerns about cultural literacy, Hirsch (1983) claimed that the standards of cultural literacy were set by the 
college admissions testing programs. Other critics have raised questions about how literature is tested in 
state and national assessments and the effects of these tests on the national curriculum (Applebee, Langer, 
& Mullis, 1987; Ravitch & Finn, 1987). In the recent review of tests (Brody, DcMilo, & Purves, 1989) it 
was shown that, in general, literature as a school subject was not tested-neither for the knowledge 
component nor for the skills and abilities related to reading a text. This problem appeared to stem in part 
from the limitations of the multiple-choice test. 

The history of the testing of learning in literature, however, has shown a great deal of variety and 
ingenuity in measurement (Purves, 1971; Purves & Beach, 1972; Cooper 1985). Many approaches to 
measuring student understanding and appreciation of literature have been studied, as have aspects of attitudes 
and beliefs. In general, these have not been explored in an attempt to make a comprehensive assessment of 
student learning. Usually measures of knowledge are divorced from measures of critical reading, and both arc 
separated from measures of learned attitudes and beliefs. Some tests, such as the daduate Record 
Examination, bring together disparate aspects of student learning, but the relationships among tHese aspects 
are not explored. The question of what might provide a comprehensive assessment of student learning of 
literature, therefore, has been ignored. To remedy this gap in our understanding of student learning and 
achievement, the present series of studies has been initiated. 

This report begins with the establishment of a theoretical depiction of 'Jie domain of literature 
learning and then reports on a series of studies to probe the interrelationships amon3 the various sub- 
domains. 



I. Background 

The Domain of School Literature 

School literature is usually seen as one of the language arts, which have often been defined in 
terms of reading, writing, speaking and listening. Since literature involves texts that people read or write, 
and since when students read literature they often write about what they have read, literature is often seen 
simply as a subset of reading and writing, with an occasional nod to speaking and listening (neither of 
which is usually tested; and they will not be in this study). But those who take a serious interest in 
literature as a school subject are uneasy with this definition. They become more uneasy when they look at 
the world of tests and see that literature is simply a vehicle for tests of reading comprehension or for 
measures of writing skill or proficiency. There seems to be the need for something more. To define the 
literature curriculum as simply a subset of reading and writing neglects a number of the acts that go on 
within the activity of literature education. 

Some would defrie literature as a school subject that has its own corpus of knowledge. Recently, 
this corpus of knowledge hxs come to be included under the heading "cultural literacy." Narrowly defined the 
corpus refers to the names associated with a particular set of texts: authors, characters, plots, and themes. 




But it might well be broadened to include such matters as critical terms like metaphor and simile, as well as 
genres, styles of writing, and whole critical approaches. 

There is another group that would see literature as something that is not read as other kinds of 
informational texts are. Louise Rosenblatt (1978) calls this kind of reading aesthetic and opposes it to the 
reading that one does with informational texts such as those of social studies and science. From this 
approach, one sees that a part of literature education is the development of preferences: habits of mind in 
reading and writing. In addition, literature education is supposed to develop something called "taste" or the 
love of "good literature," so that literature education goes beyond reading and writing in the inculcation of 
specific sets of preferences and habits of reading and writing about that particular body of texts which is 
called literature. 

We would then argue that the domain of school literature can best be divided into three interrelated 
aspects: knowledge, practice, and preference. The interrelationships are complex in that one uses knowledge 
in the various acts that constitute practice and preferences, and practices and preferences can have their 
influence on knowledge. One can hypothesize that the intersections among the aspects can best be seen (at 
least in the United States pedagogical tradition, Purves, 1971) in the relationship between knowledge of 
critical terms and the practice of writing about texts, and in the practice of reading and writing and aesthetic 
judgments. One could argue that these judgments based on actual reading serve to steer the reading interests 
and habits of people. At the same time, one can separate them for the purposes of testing and curriculum 
planning. We may schematize the three sub-domains as follows: 
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If we accept this depiction of the domain, we see that one major question facing those concerned 
with test generation is to determine what sorts and combinations of acts, texts, and types of items might 
best be suited to measure a student's performance within and across the domain. The issue of item type is 
central to this study, for there is concern throughout the profession that the standard item-type-the 
multiple-choice or matching question-is too limited (Brody, DeMilo, & Purves, 1989). A review of 
existing tests and measures in this country and overseas produced the classification of item types found in 
Appendix 1. Against this background, the present study was designed. 



II, Design of the Present Study 



The research team as a starting point the fact that there are a variety of concerns in the 
teaching and testing community, each of which calls for a comparison of testing types. The team also 
determined that it would be impractical to attempt testing of oral skills at this point, since the main 
purpose is to propose alternatives for large-scale testing programs. It was clear that one point of the study 
was to compa ~ practice-the most frequently tested aspect of the domain-with knowledge, on the one hand, 
and preference on the other. Practice would remain the centerpiece, since by all accounts it is the most 
important aspect of the curriculum. Within the area of practice itself, there exist some measurement issues 
which could be examined in this study. One was the relationship among various modes of response: the 
multiple-choice, the short open-ended, and the long open-ended response. The short open-ended response 
mode might be seen as the pivot upon which to balance the other two. A second concern was whether there 
would be significant differences according to genres, or whether a single genre might stand as a surrogate for 
the others. Of particular concern was the relationship between literary and non-literary texts, both with 
respect to practice and with respect to aesthetic judgment Finally there has been raised a concern about the 
scoring of open-ended responses: whether the quality of the writing could be distinguished from the quality 
of the understanding of the text. 

This set of concerns led to the following matrix: 



KNOWLEDGE PRACTICE PREFERENCE 



Textual Extratextual Reading Writing Aesthetic Habit 

Item Type 
open-end 
short 
extended 
closed-end 



One should note that the concern over item type applies primarily to the practice column so that 
the comparisons of item types were to be most fully elaborated in that aspect. 

In order to begin to look at the relationships among the cells of the matrix, the team selected a 
partial cross-over design in which pairs of texts with various types of measures would be matched against 
each other (cf. Choppin & Purves, 1969). For economy of scale given the number of variables, a full cross- 
over was not employed. Because there were some types of measures which were not dependent upon a text, 
these measures were paired with measures based on the reading of a text. In this way the practices of reading 
and writing became central to the study. 



Task Specification 

Follo\ »ng the general review, the research team settled on the following sets of tasks. 




Background Knowledge 



As has been suggested above, this consists of two subsets: knowledge of specific texts and 
knowledge of critical terminology. Knowledge of critical systems and theory is less appropriate at the 
secondary level than at the college level and so was omitted. The selection of specific texts was based on the 
work of Hirsch and was limited to figures and stories from the major Western traditions of legend including 
the biblical, the Greek, the Norse, and the American. The terms selected were terms that appeared frequently 
in literature textbooks and covered poetry, fiction, and non-fiction. The items could be either short answer 
or multiple-choice and it was decided to include one measure reflecting each item type. 



Practice of Reading and Writing 

As this sub-domain involves the application of reading and writing skills to texts that presumably 
the student has not read (if they have been read and studied, the measure becomes to some extent a measure 
of knowledge), it was decided to select a variety of texts and to pose three different sorts of measures with 
respect to each one: a measure that asked brief questions concerning the content and form of the text; a 
measure that translated those questions into multiple-choice questions so as to minimize the writing 
constraints; and a task that asked for an extended piece of writing so as to maximize the writing constraints. 
It was decided that the texts should represent the genres of poetry, prose fiction, and prose non-fiction, and 
that more than one text should represent the genre. 



Aesthetic Judgments and Habits and Preferences 

Since this sub-domain includes both the judgments of individual texts and general interests and 
beliefs, one measure could be attached to each of the selected texts. This measure would ask for the students' 
judgment of the particular text. A second measure or cluster of measures would deal with reading interests 
and habits and with beliefs about literature. Each of these measures could use an open-ended or a scaled or 
selected response; the latter was selected for this study. 



The total test-package is represented by the matrix that follows: 



KNOWLEDGE PRACTICE PREFERRED HABITS 



Textual Extratextual Reading/Writing Aesthetic Habit 
open-end X X 

short X X 

extended X 
closed-end X XXX 



Text Selection 



The measures focused on six texts: "The Man by the Fountain" by Georges Hebbelinck and "The 
Use of Force" by William Carlos Williams (fiction); "The Birth of the Moon" by an anonymous author and 
"The Iks" by Lewis Thomas (non-fiction); and "Forgive My Guilt" by Robert Tristram Coffin and 
"Dandelions" by Deborah Austin (poetry). These texts were selected to provide a range of structures, 
language uses, and degrees of personal involvement on the part of the writer. Three of the texts: "The Man 
by the Fountain," "The Birth of the Moon," and "Forgive My Guilt" had been used in a prior study by the 
Center (Langer, 1989) and it was thought that eventually results of the two strands of research could be 
connected. 

"The Use of Force" is a first-person story of a doctor's attempt to examine the throat of a young 
girl and his realization of how violent he has become in assaulting her. "The Man by the Fountain" is a 
third-person story of an old man's brief encounter with a young runaway who he finally turns over to the 
police. "The Iks" is a reflective essay contrasting an anthropologist's view of a savage tribe with the 
author's own. "Birth of the Moon" is a process-oriented description of the theories surrounding the 
separation of the moon from the Earth. "Dandelions" is a humorous poem about a homeowner's battle 
against the dandelions in her lawn that relies on a single extended metaphor to make its point. And 
"Forgive My Guilt" is a reflection on a hunter's maiming and killing two birds. 



Item Generation 



For the cognitive measures associated with the texts, the questions were limited to a set of 
common topics that could clearly be related to literature education. The following topics were selected to 
provide a range of items across the literary and non-literary aspects of the domain: content, character 
motivation, over-all structure, and figurative language. 

In order to facilitate the comparisons, the team decided to use the open-ended short answer 
questions on each text as the anchor measure for each text and for all comparisons. The short-answer items 
were developed first and from them the multiple-choice and essay items were developed. (The only 
exception to this practice was with the two short stories, which had been used in an earlier study with 
multiple-choice questions already written, so that the short-answer questions were generated from preselected 
multiple-choice items; Purves, 1973). The essay questions were written to bring together two of the topics: 
structure, and either content or character motivation. 

The measures of extratextual knowledge and of preference (aesthetic, interest and attitudes) 
contained some general items as well as items specific to texts. The measure of knowledge of terms was 
generated from a list of terms that were used by students in their discussions of the texts in the previous 
study (Langer, 1989) together with parallel items for the new texts (e.g. irony). The background knowledge 
items dealing with myth were adapted from early National Assessment items (NAEP, 1973). 

The other sets of items that had already been developed and validated were the transfer and interest 
items (Purves, 1973) and the censorship items (Dwyer & Summy.1988). The aesthetic judgment items 
were adapted from the validated measure developed by Fredholm (1974), based on the major criteria used 
for making aesthetic judgments of literature. The only new set of items was one designed to elicit the 
students' attitudes towards the text, i.e., whether they considered it literary or not. In each case, scaled 
measures seemed an economical approach to a complex issue. 
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The questions were all written and reviewed by an internal jury and then sent to an external jury for 
comment and review parallel to the pilot-testing. A sample of the measures is included in Appendix 2. 

Sample Administration, and Scoring 

In the spring of 1989, the packages of tests that had been developed were administered to a sample 
of upper-level secondary school students front urban, suburban, and rural schools in the state of New York. 
The schools were asked to supply eleventh-grade students, but in some cases tenth- or twelfth-grade students 
were also tested. Teachers were asked to test the full range of students in the grade. The designed sample 
called for a minimum of 30 students per item type, but because of absenteeism and the fact that certain 
classes were not tested, the achieved sample was as in Table 1. The poor sample means that reliabilities are 
not high and thus that any results must be treated as suggestive rather than as definitive. The findings 
should be seer, to comment on the test and scoring characteristics rather than on the characteristics of 
students or groups of students. 

TABLE 1 
Designed and Achieved Samples 

Measure 

Short Ans Multiple-Choice Essay Aesthetic 

Ach/Des Ach/Des Ach/Des Ach/Des 



Text 

Story 1 154/210 20/30 15/30 26730 

Story 2 122/180 19/30 22/30 26/30 

Poem 1 145/210 19/30 18/30 21/30 

Poem 2 119/180 25/30 19/30 24/30 

Non Fiction 1 86/180 25/30 25/30 18/30 

Non Fiction 2 88/180 23/30 24/30 18/30 

Attitude/Interest 113/120 

Censorship 60/120 

Myth 112/120 

Terms 60/120 
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Arrangement of Packages 

The packages were so arranged that each student took a short-answer test on one of the six texts 
followed by one of the other tests. The design is such that the short-answer test becomes the anchor against 
which all other measures are judged. The design of the packages is as in Figure 1. This is an incomplete 
cross-over design, as the design had to be limited to the size of the sample. Thus there is no comparison 
within type of measure and type of text, only across measures and text-types. The packages were rotated in 
a class so that no more than six students in any class took one specific pairing. In this way the tests were 
distributed across the population. In general the testing time was one normal class period, which averaged 
45 minutes. The sections were not separately timed, in some instances the time allotted proved to be too 
short for some of the pairings, particularly those which included an essay, as can be seen in the number of 
blank responses. 



Figure 1 
Design of Packages 



Student 



Story 
1 2 



Poem 
3 4 



Non-Fiction 
5 6 



1 Short Answer 
Multiple Choice 



A B 
B A 



C D 
D C 



E F 
F E 



2 Short Answer 
Essay 



A B 
B A 



C D 
D C 



E F 
F E 



3 Short Answer 
Aesthetic Judgment 



A B 
B A 



C D 
D C 



E F 
F E 



4 Short Answer 
Attitude/Background 



A B 
B A 



C D 
D C 



E F 
F E 



5 Short Answer: Fiction A B 
Poem C D 

Non-Fic 



A B 



E F 



C D 
E F 



The short answer questions were scored on a four-point scale (blank, poor, average, excellent) by 
one rater with every tenth response scored by a second independent rater. The scoring rubric was based on 
the principle of appropriateness and amplification of the answers. A one-word response was usually 
inadequate; an excellent answer gave one or two supporting details or an amplification of the generalization. 
The percentage of exact agreement of the raters was 88%. The essay responses were scored on the same 
four-point scale by two independent raters, both doctoral students in language education. One rater scored the 
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compositions for "literary acumen" basing the score on the accuracy of the response in terms of its 
appropriateness to the text and an interpretation that could be supported with references to the text; the 
second rater scored them for "writing ability," based on structure, organization, and stylistic appropriateness. 
Both scores were also check-scored by a third judge, with exact agreement on 70% of each rating of the 
compositions. 



III. Results 



Individual Test and Item Characteristics 

Despite the low response rates in multiple-choice and essay tests, there is sufficient information to 
indicate that the other measures performed well (Tables 1-21, Appendix 3). The various passages were 
roughly similar in their difficulty levels on the short-answer (6 out of 15 pairs were significantly different), 
multiple-choice (no significant differences) and essay scores (4 out of 30 pairs reached significance). 
"Dandelions" proved the most difficult on the short-answer and "The Birth of the Moon," the easiest (Table 
IB). The metaphor of "Dandelions" proved a difficulty and "The Birth of the Moon" is a straight-forward 
informative piece. The range of item difficulty in the multiple-choice measures was less consirtent than on 
the short-answer measures; there were two items on which all or nearly all selected the key and one on 
which fewer than 20% selected it. In general the essay score distributions for both writing and literature 
(1.7 to 2.6 on a 0-4 scale) subscores appear appropriate, but the standard deviation was affected by the 
number of blank responses which were scored as zero. 

The aesthetic judgment measure appeared to possess adequate psychometric properties. Since the 
students were asked to rate each of sixteen different aesthetic judgments, it was not possible to do a factor 
analysis on each selection. Instead, they were polled for each genre and then combined across all genres. On 
the basis of the factor scores, it was thought possible to derive a measure that used fewer questions to gain 
similar information. The results (Appendix 3, Tables 7-14) show that clear factor scores ca.i be derived, but 
that these are only interpretable as negative and positive ratings, rather than as factor scores indicating the 
particular nature of the criteria selected (e.g. emotional, aesthetic, or meaning-based). This result appears 
unrelated to the absolute rating of the text (Appendix 3, Table 15) which indicates th*. all of the texts were 
viewed more positively than negatively. It also contradicts the results of the most frvrved criteria measure 
(Appendix 3, Table 21) which indicates that the genre appears to affect the criteria elected regardless of 
whether the text is judged good or bad. Such a finding suggests that the measure needs revision so as to 
distinguish the judgment of quality from the nature of the criteria selected if one wants to find out both 
sorts of information. 

The text attitude measure was one which asked the degree to which the students viewed the text as 
"literature" or as "school literature." The results (Appendix 3, Table 16) support the findings of the aesthetic 
criteria measure that students generally view poetry as literary and so judge it, and non-fiction as non- 
literary and so judge it. When they rate fiction, however, they appear to mix literary and non-Mterary 
judgments. The reasons for this might be explained by the nature of the genre, but it is more probable that 
they occur as a result of way in which fiction is taught and tested, for it tends to be seen as a hybrid 
between aesthetic and instructional text (Brody, DeMilo, and Purves, 1989). The general attitude and interest 
measures (Table 19) showed results similar to those found in earlier studies (Purves 1973): two factors of 
interest emerge, a generalized interest in reading and literature with a willingness to participate, and a 
tendency to project oneself into what one reads as opposed to a tendency to remr.in distant and aloof. The 
censorship measure also showed two broad factors (Table 20): a tendency to censor on multiple grounds, and 
a tendency to restrict censorship to pornography. 

The two background measures appear to have been psychometrically well-constructed (Tables 17 
and 18), producing an expected mean and a standard deviation that would appear to reflect instruction as well 
as time for response. None of the items was particularly easy or particularly hard except for the item calling 
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for a definition of exposition. The students defined it in rhetorical terms as a type of composition rather 
than in literary ones as an aspect of narrative plot. 

One can conclude from these results that the individual components of the measures performed well 
in the trials, although the numbers of responses for the multiple-choice and the essay responses were 
insufficient to make any but the most general comments. The separate cognitive measures appeared to be 
well constructed with a mix of easy and difficult questions, and few questions that should be rejected from a 
final version. The attitude measures also seemed to be appropriately constructed, with the exception of the 
aesthetic judgment measure which proved difficult to interpret. Whether these individual types of tests and 
scales measured discrete or confounded aspects of the domain of literature achievement remains the next 
question. 



Comparative Results 

Because of the small samples for some of the text-related measures, it was decided to treat each of 
the text types and test types as a unit for purposes of comparison. Thus the individual story, poem, and 
non-fiction responses for short answer, multiple-choice, and essay scores respectively were aggregated across 
the genres for the purposes of making correlations. This was done by converting the scores into standard 
scores and then adding across genres and basing the correlations upon the standard score sums for each genre. 

The correlation between the open-ended and multiple choice measures can be seen in Appendix 3, 
Table 22, and the inconsistent results do not appear to support the findings of Choppin and Purves (1969), 
but one cannot determine whether this finding results from text, measure, scoring, or sampling effects. A 
similar in insistency can be seen in the comparison of the essay and short-answer measures (Appendix 3, 
Tables 23 and 24). The writing and literature essay scores are highly correlated, but a part of that overlap 
resulted from the failure of the two scoring schemes to differentiate clearly the criteria they were using. 
Interviews with the raters showed that they tended not to follow the explicit rubric. A small-scale rescoring 
effort indicated that the correlations would drop to about .6, suggesting that the bi-partite scoring makes 
sense as long as the scoring rubric is made clear to each scoring team. 

The relationships between the measures of knowledge and practice are generally positive but not 
overwhelmingly so (Appendix 3, Table 25). It is worthy of note that knowledge of terms is a better 
predictor of measures of understanding than knowledge of culture and myth, a finding that would not 
support the contention made by the proponents of cultural literacy (Hirsch, 1987). This relationship clearly 
needs further exploration, perhaps to determine whether terminology is taught more consistenUy across 
schools than are individual texts. 

The relationships among the various aesthetic judgments of the text and understandings of the text 
are virtually non-existent or so mixed that one must conclude that students generally do not let their rating 
affect their understanding or vice-versa (Table 26). This finding contradicts the results of other research 
(Purves, et al 1980), which suggests that students who do not understand a text tend to blame the text and 
thus indicate their dislike of :i Such mav he the case here, but there is insufficient evidence to be sure. 

When one examines the relationship between practice and general attitudes towards literature (Table 
27), one finds that students who like literature perform better, but that transfer appears an independent 
construct. It is worthy of note that the students who read texts well disapprove of censorship more than 
those who read them poorly. This result supports the findings which showed the negative correlation of 
verbal ability and disapproval of censorship. 

The final comparison is that among the text types themselves on the short answer measures 
(Appendix 5, Table 28). The resuiis indicate that performance on one text will predict performance on 
another but that the strength of the prediction (i.e. cuTclations ranging from .41 to .51) is weak enough to 
suggest that one should use more than one text to get a reliable and valid estimate of student literary 
understanding. 
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The basic comparisons may be summarized as follows (Table 2): 



Table 2 

Summary nf Correlations of Short- Ans wer Scores Wit h Other Scores 





Story 


Poem 


Non-Fiction 


Multiple-Choice 


.10 


.37 


.53 


Essay 


.53 


.27 


.25 


Myth 


.25 


.26 


.29 


Terms 


.18 


.49 


.69 


Rating of Text 


.25 


.27 


.25 


Transfer 


.27 


-.05 


.05 


Interest 


.22 


.13 


.26 


Censorship 


-.26 


-.02 


-.41 



As was stated earlier, the short-answer measure becomes the anchor against which the other 
measures are compared. The comparison can be extended across each of the three genres. One might set a 
con elation of .7 as the point above which two measures may be said to be redundant. This figure is 
approached only once, in the correlation of knowledge of terms with the reading of non-fiction. There are 
three correlations that are close to chance (between -.05 and .05). The relative independence of the transfer 
score from achievement suggest that it might be a psychological penchant independent of any other aspect 
of literature learning, but there is not strong enough evidence to suggest that the measure should be 
eliminated from a comprehensive package. In summary, because of the small sample and low reliability, 
none of the correlations is sufficiently strong to suggest that any of the measures is clearly measuring the 
same construct with respect to literature and the reading of literature although the possibility remains. The 
variation across genres also suggests that the three genres are not redundant from the perspective of 
measurement of achievement in the domain of literature learning. 



IV. Conclusion 

Because of the relatively small sample, it is difficult to reach strong conclusions about student 
behavior and understanding. Treating the study as a pilot-testing exercise, which, to a great extent, it was, 
we can reach some clear conclusions as to what might make a good comprehensive test package. The 
results do tend to validate the model that knowledge, practice, and preference are related but not highly 
interrelated aspects of the construct of literature learning. A comprehensive measure of student performance, 
therefore, should address each of the three areas. It would appear tbat within the knowledge domain, textual 
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knowledge and knowledge of critical terms are distinct, particularly in their relationship to the practice of 
reading and responding. Within the domain of practice, more than one passage is needed to get some 
estimate of a student's performance across text-types. It seems to make little difference whether one uses 
open-ended or multiple-choice questions, but one can argue on other grounds that open-ended questions 
probably present somewhat more of a challenge to students than multiple-choice questions (Hansson, 
1990), and would therefore be a more exacting measure of the ability to read and shape a response to what is 
read. 

It is also clear that an extended response is also desirable, but other studies argue that the phrasing 
of the question might be such as to allow the student some preparation for the setting forth of a fully 
articulated composition. A stark question is less desirable than a question that builds upon another sort of 
task, one that gets the student to consider the text in question (Hansson, 1990). A combination of multiple- 
choice and essay or scale and essay might be the optimum measures. 

In the realm of preference, » . would appear important to separate determining the student's criteria 
for judging a text from the actual judgment. It would also appear to be important to get a depiction of the 
general attitudes towards literature including censorship, since these clearly appear to be related to cognitive 
performance (whether in an antecedent or consequent role remains unclear). 

From these conclusions one may derive a set of specifications for an assessment of student 
learning in literature that would include the following: 

1 . Measures of Background Knowledge-terminology and cultural information; these may 
include matching and supplying or generating items 

2. Measures of the ability to read and articulate a written response to at least two texts 
that differ in genre, the measures to include both supplying and constructing items, with 
the latter taking the form of extended discourse. 

3. Measures of preference including aesthetic judgment of specific texts and general habitr 
and beliefs concerning literature and its place in the world. 

Such an assessment would give a more comprehensive picture of student learning and also of 
program effectiveness than would a measure of any one taken alone. A recent study that used a more 
comprehensive measure showed that a complex measure served best to validate a model of instruction. If 
the intention of the instruction is to make classroom exploration of literature more open and to use more 
"real" and thought-provoking questioning than normal instruction, its validation must include measures of 
both practice and preference (Ho, 1987). It is the trial of such an assessment package that will form the next 
phase of this area of research. If successful, the package might well serve as a model of assessment at the 
state cr district level, and at the classroom level. 
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Appendix 1 



A Note on Item Classification 

One of the most common ways of classifying test items in literature is by the content of the item, 
whether and in what way it refers to the text. Another common way has been in terms of the format, 
whether short answer or multiple choice. A third has been according to the Taxonomy of Educational 
Objectives, a project of the 1950's that attempted to define items in terms of the cognitive skills they called 
for (Bloom, et al., 1954). This last way has proved somewhat controversial in that the project discriminated 
between the cognitive and affective domains and attempted to establish a mental hierarchy. Assessment 
projects at the Center for the Learning and Teaching of Literature have been examining literature tests from 
the perspective of content and cognitive skills (Brody, DcMilo, and Purves, 1989), with success on the first 
perspective and less so on the second. 

It seems that test questions might best be examined in terms of a combination of format and 
demand upon the student. Such an approach parallels the work on composition tasks performed by 
Vahapassi (1988) who saw them as an interaction of cognitive demand and discourse function. With respect 
to tasks related to the reading of literature, this combination would enable one to see that the task 
confronting the student, for the most part, has to do with the degree to which the student must supply new 
information rather than use information that is present. This degree is related to the degree to which the 
student must assimilate the particular text and come to conclusions about it, which conclusions must then 
be articulated. Such an approach is related to that set forth in the work of Hansson (1990). It is clear that in 
an essay question, for example, a student must supply information in order to articulate a response, drawing 
upon long-term memory and to some extent generating new information and new structures. In a multiple- 
choice question, on the other hand, the student must look at the information presented and select the best 
match between stem and options. In a literature test the student mav also need to have recourse to the text, 
but the student need not generate an articulated interpretation of the text in order to get the "right 1 ' answer. 
From these two extremes, we may perhaps develop an item classification system that will be of use to 
those constructing and analyzing tests in literature which involve knowledge, practice and preference. It may 
be too that there are implications for the curriculum. 

I Items where all information is present or available. 

A. Matching 

i. Picture-text 

ii. Tex^-text 

iii. Interpretation-text 

B. Sorting 

i. Pictures 

ii. Words/phrases/sentences 

iii. Continuous text 

C. Reorganizing 

i. Pictures 

ii. Words/phrases/sentcnces 

iii. Continuous text 

II Items where the majority of information is present or available, the rest is presumed to be in the 
long-term memory and is to be applied to the present situation (cf. Broudy 1982). 
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D. Supplying 



i. Pictures 

ii. Words/phrases/sentences 

iii. Continuous text 

III Items where the information was present in a previously read text and is presumed to be in the 
long-term memory. The primary task is to make a report of what is known, usually within a 
prescribed form. In Broudy's terms the use of learning is primarily replicative. 

E. Recalling 

i. Words/phrases/sentences 

ii. Continuous text 

F. Paraphrasing/Summarizing 

i. Words/phrases/sentences 

ii. Continuous text 

IV Items where the information is not presumed to have been directly present in the long-term 
memory but in which the student is required to make an interpretive or associative use of prior 
learning (Broudy, 1982). 

G. Constructing/generating 

i. Words/phrases/sentences 

ii. Continuous text 

iii. Interpretation 

iv. Synthesisfludgment 
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Sample Measures 
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The Man by the Fountain 



As always, John Deweck sat by the fountain. 

The spring sun loomed up out of the seething foam. The children honoured the memories of 
heroic admirals. Their galleons and cutters tacked to and fro across the wide pond. Nursemaids and 
grandmothers glanced anxiously at frocks and trousers. Over the wide world the fountain sang, thrusting a 
quivering plume of water at the scudding clouds. Liquid pattered noisily into bowls of marble. 

John Deweck sat on his usual bench, speaking to no one. There were a few rules he stubbornly 
clung to. People spoke so much ill of each other. He no longer listened to their chatter. He had eyes now 
only for students and soldiers, for young girls and children. Young people fascinated his old carcass. He 
knew a great deal and had forgotten even more. He craved for youth and approached death's kingdom with 
reluctant steps. 

One by one the frequenters of the fountain left the park. It was time for lunch. John smiled 
without quite knowing why. Now that he was alone, it seemed to him that he was the head park keeper. It 
was Thursday. The day on which his wife always used to serve him veal-steak with a delicious sour sauce 
and potatoes as round as marbles. She had been able to work miracles with a potato. Since her death he 
had fallen into irregular eating habits. Three slices of bread and jam in the morning. At midday, often not 
even a bite. Round about five, some lumpy porridge with rusks and some fruit. Usually a sour apple. Sour 
apples, he believed kept the mental juices clean and preserved understanding. 

He sat now alone with the violence of the fountain. 

Perhaps some little boy would turn up? He longed for a serious conversation. Eyes that were 
still keen swept the avenue that led to the outskirts of the town. Far off in the distance, as in a dream, the 
little boy came into view. 

The youngster came tearing up to him, flopped down on the bench and gazed spellbound at the 
rippling surface of the pond and at the dragons letting the water flow over their green breasts. 

"Hello, young man," said John Deweck solemnly. 

The child stared at him but said nothing. 

"Isn't it your dinner-time?" 

"I'm not hungry." said the boy. "I eat once a day. Raw buffalo-meat, as I roam the praiwe on my 

bronco." 

"Well, now," said John Deweck, "Well now. . . who might you be then?" 
The boy looked up at him full of pride. 

"I am the last of Uu Mohicans. I lest my friend-the paleface. He was caught in an ambush. But 
I scented danger. Now I wander alone through the wood and valley. . .." 

"Where are your feathers?" asked old John sternly. 

The child gazed at him with lively interest. Tiny flames flickered in the golden eyes. He flushed 
with excitement. 

"I don't wear feathers in enemy country," he said in a whisper. "But still, I'm on the warpath. I've 
no war p->int on but I've dug up the hatchet. I am the last of my tribe. Are you my friend or foe?" 

"What a thing to ask! My name is John. I have always been the foe of the buffaloes and the friend 
of the Indians. I made a blood-pact with Winnetou. Now I am too old for the hunt. Against whom have 
you dug up the hatchet?" 
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"Against the tribe of grown-ups," answered the boy. "They threaten my hunting-grounds and my 
freedom. They don't understand a thing. How can an Indian live in stuffy school-buildings?" 

"Of course he can't," said John. "Though a paleface myself, I'm all for freedom, too. But still, I 
think school is necessary. . ." 

The youngster direw him a piercing look. 

"Perhaps you're a spy," he said thoughtfully. "The enemy is cunning." 
John Deweck gave a high-pitched laugh. 

"Nonsense. Take a look around. We're quite alone here. No, I'm not a member of the tribe of 
grown-ups." 

"How strange. So old, yet still a good Indian." 

The old man gave a loud sniff. He held his hand out to the young brave. 

"Peace," he said, "and many scalps." 

"I'll tell you my adventure," said the boy, "provided you can keep a secret." 
"Even if I was bound to the torture-post I wouldn't breathe a word." 

"This morning I had to hunt for buffalo. As you know, the time has come. Besides, I'm looking 
for a squaw for my new wigwam. I was creeping out of the kitcher when Dad caught me by the hair. He 
walloped me for not being ready for school. I didn't make a sound. Only cunning could save me. Meekly I 
let myself be led to Hook Nose." 

"Who is Hook Nose?" 

"The school chief," replied the boy. "He's not strong but he's terribly cunning. He laughed like a 
wild horse and spoke of giving me lines. At ten o'clock, during break, I sneaked out at the gate. I ran as 
fast as I could. . . I don't want to go home again. My homeland is the prairie. Tonight I'm looking for a 
boat and tomorrow I'll be sailing across the seas." 

John Deweck looked at the fountain. Impetuously as life itself ii leapt up towards the light of the 
boundless sky. Cherubs spattered with water, blew on their conches as if to warn of impending danger. 

A wrinkle creased the aged forehead. 

"It's not going to be an easy plan," sighed John Deweck. 

"I must get a boat," said the boy stubbornly. "You've got to help me." 

Heavy clouds drifted towards the spring sun. The birds were silent in the pruned trees. 

"First come and eat in my wigwam" faltered John Deweck. 

Tin not hungry." 

"You can't refuse bread and salt. . ." 
The boy thought this over. 

"Your mouth speaks the truth," he said. "I must set out on my long journey free from hunger. 
But I shan't eat meat" 

"Bread and salt, O warrior. . ." 

The boy trotted at the old man's side, locking neither left nor right. He thought of the wild scents 
of the prairie. He had met an old buffalo-hunter who gave him invaluable tips. 

They stepped into the police station. The door closed behind them with a bang. The boy looked 
about him and understood 

He sat down on a bench and freely volunteered information to a fat man with a ruddy complexion. 
His head sank on his chest He did not even glance at John Deweck. 
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The car arrived shortly afterwards. The father stepped out and thanked the old man. The boy took 
his place in the car. Suddenly, he turned to the buffalo-hunter. 

H You belong to the tribe of grown-ups/ 1 he said. "You have betrayed my confidence. I will pay 
for it at the torture post. I despise you." 

He spat on the ground. 

"What did he say?, asked the father 

"That you ought to make him happy," said John De week. 

Father and son vanished in a cloud of dust 

"The youth of today," grunted the inspector. 

Slowiy the old man paced through the streets of the little town. 

He was never seen again at the fountain. 

George Hebbelinck-Bclgian 

[Reprinted with the permission of the International Association for the Evaluation of Educational 
Achievement (IEA).] 
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THE MAN BY THE FOUNTAIN (short answer) 

For each of the following questions write out your best answer in the space that follows. 
1. How is the fountain first described in the story? 



2. How is the description of the fountain in lines 76 to 78 different from the initial description in 
lines 2 to 7? 



3. How do the boy's feelings toward John Dewcck change in the story? 



4. How does John Deweck's life change at the end of the story? 



5. What does the description of the fountain in lines 76 to 78 contribute to the story? (What does 
the description convey?) 
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6. Which descriptions following the description of the fountain in lines 76 to 78 reinforce the 
mood that is established there? 



7. How does the boy's view of his life and of John Deweck change in lines 91 to 95? 



8. What do you think the descriptions of the fountain - all viewed together- contribute to the story? 
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THE MAN BY THE FOUNTAIN (multiple choice) 



For each of the questions that follow, select the one answer that best matches your understanding of 
the text. Circle the letter that corresponds to your choice. 

1. Three of the following are contained in the description of the fountain in lines 2 to 7. Choose 

the one that is not, 

a. It is intermittent 

b. It is pleasurable. 

c. It is active. 

d. It is wide-spreading. 

2. Which of the following words or phrases in lines 76 to 78 present a different aspect of the 

fountain from what was described in lines 2 to 7? 

a. "Impetuously as life itself 1 (line 76) 

b. "impending danger" (line 78) 

c. "spattered with water" (line 77) 

d. "it leapt up" (line 76) 

3. In the paragraph beginning on line 91, which of the following would be an accurate description 

of the boy s feelings about John Deweck? 

£L Trust and admiration. 

b. Nervousness and fear. 

c. Loathing and detestation. 

d. Tolerance and approval. 

4. There are three possible explanations of the last sentence in the context of the story as a whole. 

Choose the explanation that is NOT possible. 

a. John Deweck realized that he would always be lonely. 

b. John Deweck felt sorry for what he had done. 

c. John Deweck wanted to live with the boy and his family. 

d. John Deweck realized that he could not recapture his youth. 

5. Lines 71 to 75 may be said to be one of the points at which the direction of the story turns. 

Which of the following best summarizes the turn? 

a. The boy decides that he must go back to school. 

b. John Deweck decides to make the boy uneasy. 

c. The boy realizes that John Deweck is not his friend. 

d. John Deweck realizes that he must take the boy to his parents. 
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6. Which of the following words or phrases most clearly reinforce (s) the change that comes after 
line 79? I. "Heavy clouds drifted"(line 82). II "The birds were silent" (line 82). II "looking 
neither left nor right" (line 91). 

a. I only. 

b. Ill only. 

c. I and II only. 

d. II and III only. 



7. Which of the following statements best describes the relationship of the two paragraphs (lines 

91 to 95) to each other? 

a. The first describes events before the story began, the second describes the results of those 

events. 

b. The second presents a reality that contradicts the dram of the first* 

c. The first describes the old man, and the second describes the boy. 

d. The second explains the boy's actions that arc described in the first 

8. Which of the following statements about the significance of the fountain is most consistent 

with the story as a whole? 

a. It suggests the liveliness of the world and of youth* 

b. It suggests the mystery of life and the pleasure of old age. 

c. It suggests the perils that face people who travel. 

d. It suggests the happiness that we lose when a loved one dies. 
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FOUNTAIN (essay) 



Write a composition of two or three paragraphs in which you compare the different images related 
to the fountain and their effect upon your understanding of the story, "The Man By the Fountain." Your 
composition will be judged on the quality of your answer to the question as well as on its organization, style 
and use of the language. 
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Indicate the extent to which you agree with each of the following statements people have made 
about "The Man by the Fountain." Circle the number that indicates the extent of your agreement. 

Agree Agree Disagree Disagree 

Strongly Strongly 

1. "The Man by the Fountain" belongs 4 3 2 1 
in an anthology of literature. 

2. The subject matter of "The Man by the 

Fountain" is not appropriate to what 4 3 2 1 

I think is literature. 

3. "The Man by the Fountain" is a 

typical example of a short story. 4 3 2 1 

4. "The Man by the Fountain" is not 

written according to my understanding 4 3 2 1 

of how literature should be written. 

5. 1 would recommend "The Man by the 
Fountain" to be used in a literature course 

in my school. 4 3 2 1 

6. 1 don't like "The Man by the Fountain" 

but I think it is good. 4 3 2 I 

7. "The Man by the Fountain" uses 

language that should not be in stories. 4 3 2 1 

8. "The Man by the Fountain" is the kind 

of story English teachers like. 4 3 2 1 
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Rating Sheet 



" THE MAN BY THE FOUNTAIN 1 



Below are a number of reasons people have given for finding this story good or bad Which are 
closest to yours? Read each and circle the number that best agrees with your judgment of the story. 



Agree Agree Disagree Di sagree 
Strongly Strongly 



1. I find "The Man by the Fountain" 
good because it has taught me something. 

It has broadened my understanding. 4 

2. I find "The Man by the Fountain" bad 
because it is not captivating or exciting. 
It has not made an impression on me 
emotionally. 4 

3. I find "The Man by the Fountain" 
good because the parts are well composed, 
logical. 4 

4. I find "The Man by the Fountain" bad 
because the form and style are not 

pleasing. They don't match the content 4 

5. I find "The Man by the Fountain" good 
because symbols and metaphors art used 

in effective and meaningful way. 4 

6. I find "The Man by the Fountain" bad 
because my demands for a story are not 
fulfilled 4 

7. I find "The Man by the Fountain" good 
because the work closely follows the 



9 

I:RLC 



27 30 



traditional form and content of this 
type of story, 

8. I find "The Man by the Fountain" 
bad because it teaches something that 
I consider wrong from a personal or 
human point of view. 

9. I find "The Man by the Fountain" 
good because it is original, different, 
new and fresh, 

10. I find "The Man by the Fountain" 
bad because the author does not 
succeed in getting across what he wanted 
to say. His intentions are not fulfilled. 

11. I find "The Man by the Fountain" 
good because the text is ambiguous. It 
can be interpreted in more than one way. 

12. I find "The Man by the Fountain" bad 
because its unrealistic and false; people, 
events and settings are not realistically 
portrayed. 

13. I find "The Man by the Fountain" 
good because it's imaginative, inspiring. 

14. I find ' The Man by the Fountain" bad 
because the theme is irrelevant, 
unimportant It has no meaning for me. 

15. I find "The Man by the Fountain" 
good because it makes an honest impression 
and it appears to be written with conviction. 
The author's tme feelings are conveyed. 



16. I find "The Man by the Fountain" bad 
because its conception of the world and of 
man is not congruent with my conception. 
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Now review the judgments and answer the following: 

1. Please enter the numbers of the six judgments that matter most to your judgment of "The Man 
by the Fountain" 



2. From the six judgments that you selected in question 1, choose the two that are most similar to 
your opinion. 



3. We would like to know how you personally compare this story to other stories you have read. 
If you think it is one of the best stories you have read, rate it +3. If you think it is one of the 
worst you have read, rate it - 3. 



Here is a scale: 
One of 
the best 

+3 



good 
+2 



fairly 
good 

+1 



fairly 
poor 

-1 



poor 



-2 



One 
the worst 
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ATTITUDE (Multiple choice) 



For each of the questions that follow, select the one answer that best matches your experience and 
beliefs. Circle the letter that corresponds to your choice. 

1. Have you done something you would not ordinarily have done because you read about it in a 
story, poem, or play? (For example, when you were younger, have you dressed up as a pirate 
because you read a story about pirates?) 

a. Often 

b. Occasionally 

c. Once or twice 

d. Never 

2. While you were reading a book have you thought of yourself as one of the people In it? 

a. Often 

b. Occasionally 

c. once or twice 
d* Never 

3. Have you compared a person you met in real life with people you have read about? (For 
instance, have you ever called a strong person Samson?) 

a. Often 

b. Occasionally 

c. Once or twice 

d. Never 

4. Have you been in a situation and asked yourself what some person in a story you read would 
have done in that situation? 

a. Often 

b. Occasionally 

c. One ^ or twice 

d. Never 

5. When you read a novel or a story, do you imagine that what is happening in the story takes 
place in some town or city that you have seen? 

a. Never 

b« Once or twice 

c. Occasionally 

d. Often 
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6. Have you done something or gone somewhere, felt that this has happened before, and then 
realized in fact it happened in a book you read? 

a. Never 

b. Once or twice 

c. Occasionally 

d. Often 

7. When you read a story, how often do you imagine that the people in the story look like people 
you know? 

a. Often 

b. Occasionally 

c. Seldom 

d. Never 

8. When you meet a new person, how often do you compare the person to someone you saw in a 
movie? 

a. Often 

b. Occasionally 

c. Seldom 

d. Never 

9. How often do you think that the people you are reading about in a story are real people and not 
simply people in a story? 

a* Never 

b. Once or twice 

c. Occasionally 

d. Often 

10. When you read a story or a play, do you try to remember something that happened to you that 

is like what you are reading about? Do you say to yourself "Something like this happened to 
me once"? 

a. Never 

b. Once or twice 

c. Occasionally 

d. Often 
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1 1 . How many books have you read for your own pleasure in the past year? 

a None 

b. Fewer than 5 

c. 5 to 10 

d More than 10 

12. During the past year, how many plays have you read for your own pleasure? 

2l None 

b. One or two 

c. 3 to 5 

d More than 5 

13. During the past year, how many novels have you read for your own pleasure? 

a. None 

b. One or two 

c. 3 to 5 

d. More than 5 

14. During the past year, how many biographies have you read for your own pleasure 9 

a. None 

b. One or two 

c. 3 to 5 

d. More than 5 

15. When you choose a story or novel to read, which one of the following is most likely to be the 
reason for your choice? 

a. Friends or parents recommend it 

b. I have read other books by the same author 

c. The tide attracts me 

d. I just choose any 

16. How often do you re-read novels* stories or plays? 

a. Never 

b. Once or twice 

c. Occasionally 

d. Frequently 
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17. Have you ever gone to a movie because you Tead the story in a book? 

a- OfteL 

b. Occasionally 

c. Once or twice 

d. Never 

18. Have you ever read a book because you saw the story in a movie? 

a. Never 

b. Once or twice 

c. Occasionally 

d. Often 

19. Have you ever read a book because you saw the story on television or heard the story on the 

radio? 

a. Often 

b. Occasionally 

c. Once or twice 

d. Never 

20. After you have seen a play or movie, would you want to read a criticism of the work? 

a. Often 

b. Occasionally 

c. Once or twice 

d. Never 
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You are going to be asked a question about a passage based on a famous story that has been 
disguised in a modem form. For each of the questions that follow, select the one answer that best matches 
your understanding of the text Circle the letter that corresponds to your choice. 

Study the example below. 



EXAMPLE: 



As she walked through the strange part of town, she was a little nervous. Then an older man, 
nice-looking but a little tough, stopped her. "Where are you going, little girl?" he asked. "To 
take these books to my aunt on South Street," answered the little girl. But she would not let 
him go with her; her mother had told her to watch out for strange men. 



Which one of these characters do you think the passage is based on? 



a. Cinderella 

b. Goldilocks 

c. Sleeping Beauty 

d. Little Red Ridinghood 



The passage is like the story of Little Red Ridinghood, so you would choose (d) Little Red 
Ridinghood. 



1. Old Petersen had had a good life, a good farm--the richest in the country, a happy family, a nice 
tidy income. Even so, he wasn't proud, just thankful. Then the locusts came and ate his wheat, 
someone poisoned his wells, and to cap it off, his children died of diphtheria. What happened, 
Petersen wondered, what had he done? It was enough to make a man lose his faith. 

a. Job 

b. Exodus 

c. Barabbas 

d. The Fall of Man 



2. How would they ever recapture the fort from the Apaches? They had tried everything - raids, 
mass attack, cutting off the water - but nothing had woriced. Then Sergeant Gottschalk had an 
idea. He put a false floor in a Conestoga wagon, hid ten of his best men under it, loaded the top 
with supplies, and drove right into the town. "We're letting you have the fort," he told the 
chief, who watched the troop pull out That night, however, the ten men sneaked out and 
opened the gates and the Indians were surprised to see the troop returning. 

a. Camelot 

b. Jericho 

c. The Trojan Horse 

d. The Wolf in Sheep's Clothing 
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3. Orville came into the room and told his mother, "Now that I've finished high school, I'm going 
on to learn everything about everything; science, history, music - every-thing. I don't care 
how I get this knowledge; just so I get it. Then I'll know everything. I'll have the power I 
want." 

a. Faust 

b. Hamlet 

c. Job 

d. Macbeth 

4. Charlie whistled as he thought about it. Everything had gone right for him after he'd left 
Readville, on the other side of the divide, just bee ause he'd been warned by the gypsy he was in 
for it. He'd come into Melrose and just happened to stop the gang that had been terrorizing the 
town. So they made him sheriff. He'd married the mayor's widow, a lovely woman, even if 
she was older. "Yes," he thought, "I guess that old gypsy was wrong. I've sure found happiness 
not trouble." 

a. Joseph 

b. Oedipus 

c. Jason 

d. Saul 

5. Ernie walked into the dance not sure what was going to happen. His crowd had fought with the 
group that was giving the party but he was in a mood; his girl had just left him. He was 
spoiling for something. He looked across the room, which was smokey and dark. He saw her 
and knew this was it. He walked over "Hi. Who are you?" She looked up and he could see the 
light in her eyes was like his. "I'm Linda." 

a. Venus and Adonis 

b. Romeo and Juliet 

c. Tristan and Isolde 

d. Pyramus and Thisbe 

6. Sam had had such high hopes, such trust that all was well here in the fruitful valley. To him, 
everything seemed good. Then, somehow, something went wrong, something spoiled it. It all 
started when that fancy salesman came in and got Sam's wife to buy his encyclopedia. 

a. Job 

b. Lot's Wife 

c. The Fall of Adam 

d. The Flight into Egypt 
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7. He hadn't been sure before, but now he was. His little brother, Sid, was the one who was going 
to be the success, he was going to get everything. It wasn't that Sid was better or brighter or 
anything, just luckier. And this made Sam mad, so he wanted to kill his little brother. And one 
day he did 

a. Cain and Abel 

b. Moses and Aaron 

c. Caesar and Brutus 

d. Damon and Pythias 



8. Sean didn't know what to do. His father had died mysteriously - was it cancer? Now his mother 
was about to marry Uncle Ed. It was so sudden. Scan was suspicious. But he needed proof. 

a. Julius Caesar 

b. Macbeth 

c. Hamlet 

d. King Lear 



9. Elaine had been waiting * how long was it? Eddy had been gone twenty years ■ first in Vietnam 
and then touring through the South Pacific. Each time he'd been about to come home 
something came up. Meanwhile, lots of guys had been trying to get her to declare herself a 
widow and marry them. She'd tried every trick to stall them. She'd heard Eddy was on his way 
home. 

a. TheAeneid 

b. The Odyssey 

c. The Ring Cycle 

d. The Upanishads 



10. OUie was worried. He'd been put in charge of the whole community but he didn't know what 
to expect. Then he learned that if he gave up his reading glasses he could see into people's 
minds even if he could never read again. He thought and thought. "This would be good for 
my people/ even though it's a sacrifice. 



a. 



Vish.iu 



b. 



Buddha 



c. 



Zeus 



Odin 
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1 1. The planet was doomed. Nick knew it but he also knew others didn't care. So he decided to try 
to save as many people and animals as he could and colonize the next galaxy. Everyone 
laughed but he went ahead. 

a. Arthur 

b. Oedipus 

c. Abraham 

d. Noah 



12. Rose was a master calculator. She could figure out a balance sheet in less than an hour given 
the raw numbers. The company thought she was indispensible. Then the new director 
brought in a computer expert. He challenged Rose to see who could do the books fastest and 
most accurately. It was a close race but she won - at what cost? 

a. Pocahantas 

b. Annie Oakley 

c. John Henry 

d. PaulBunyan 
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Indicate the extent to whic* you agree with each of the following statements people have made 
about Censorship. Circle the number that indicates the extent of your agreement 



Agree 
Strongly 



Agree Disagree 



Disagree 
Strongly 



1. Librarians in elementary schools 
should remove books from circulation 
that they believe would be harmful to 
children if read by them. 

2. Clergy (priests, ministers, rabbis, etc.) 
should not have the right to remove books 
from libraries even if those books are 
offensive to the people in their places of 
worship. 

3. Books by Karl Marx should be banned 
from libraries because they glorify 
communism. 

4. Books by convicted criminals serving 
time in prison should not be banned from 
libraries even if the criminals will get rich 
from the sale of the boo*s once they are 
released from prison. 



5. Pornographic material should be 
banned from all libraries and bookstores. 
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Agree 
Strongly 

6. Teachers and school administrators 
should not be allowed to take books from 
students even if they believe the books 
contain information harmful to the 

student. 4 

7. The production of pornographic 
books should be a serious crime and the 
authors and publishers should be 
imprisoned. 4 

g. Faculty should not be allowed to 

prevent students from publishing articles 

in student newspapers. 4 

9. Censorship boards made up of out- 
standing community members should have 
the power to ban books which they believe 
have dangerous content 4 

10. Public and school libraries should not 
be permitted to have books on sex education 
available for reading by people under 16 

years of age. 4 

1 1 . Newspapers should report the articles 
of groups wanting to over-throw the gov- 
ernment of the United States. This is true 
even if the activities might be appealing to 
people not now involved with the groups. 4 



Agree Disagree 



Disagree 
Strongly 



IMC 



Agree Agree Disagree Disagree 
Strongly Strongly 

12. A newspaper should not report 
the last words of a convict about to be 
electrocuted if the convict said he was 
proud of the murder he committed and 
was glad he did it and that he thought 

of himself as a hero. 4 3 2 1 



13. Advertisements for membership in 

Ku Klux Klan should be permitted in 

high school and college newspapers. 4 3 



14. The United States government should 
not be allowed to pass laws controlling ad- 
vertising in children's newspapers. 4 



IS. Public and school libraries should 
not have books about drug use that can be 
checked out by children under 16 years of 
age. 



4 



16. Young people are helped in learning 
«vh*t is right and what is wrong when 
books with un-Christian ideas are kept 
from them. 4 



17. Taxpayers pay for the books that 
libraries and schools purchase; theiefore, 
taxpayers should have the right to determine 
which books are purchased and used by 
the community. 4 



18. Courts should not have the 
right to rule on whether or not 
books should be bannel from the 
community. 



ERIC 



43 



Agree 
Strongly 



Agree Disagree Di sagree 

Strongly 



19. Adults should not prevent 
teenagers from reading material 
written by people who glorify 
lifestyles that are very disturbing 

to the adults. 4 

20. A parent or group of parents 
should have the right to remove a 
book from a public library or public 
school curriculum if they find it 
offensive. 4 

21. Books that describe how to commit 
crimes or how to destroy property 

should be banned. 4 

22. Books which say that the United 
States was wrong in its involvement 
in the Vietnam War should be banned 
because these books could make people 

feel angry toward their country. 4 

23. A book should not be banned even 
if it says that the ancient Eskimo 
practice of sending old, helpless, people 
off to die by themselves in the snow is 
agoodidea, 4 

24. Librarians should be allowed to 
obtain every type of reading material 
because libraries are merely storage 
areas for books and should not limit 

what they hold. 4 



IMC 
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In a brief paragraph for each, write a definition of the following terms: 



1. Metaphor 



2. Symbol 



3. Onomatopoeia 



4. Personification 



5. Protagonist 
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6. Climax of a story 



7. First-person narrative 



8. Irony 



9. Exposition 



10. Sonnet 
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11. Rising action 



12. Minor character 



Appendix 3 
Tables 




TABLES 





TESTS AND ITEM CHARACTERISTICS 


TABLE la: 


SHORT ANSWER MEASURES 


( SCALE 0 24) 




Means 


Standard Deviation 


Story 1 
The Man by 
the Fountain 


16.67 


4.71 


Story 2 
The Use of 
Force 


16.54 


5.08 


Poem 1 

mm J 

Forgive my 
Guilt 


15.97 


4.08 


Poem 2 
Dandelions 


14.06 


3.79 


Non-fiction l 
The Iks 


14.79 


5.88 


Non-fiction 2 
The Birth of 
the Moon 


16.64 


4.42 
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TABLE lb! T-TEST FOR SHORT ANSW ER MEASURES ACROSS 6 TEXTS 

STORY POEM NON-FICTION 





1 


2 






X 




Story 1 
The Man by 
the Fountain 


• 23 


1.39 


4.93* 


2.71* 


.44 


Story 2 
The Use of 
Force 






1.55 


4.28* 


2.29* 


.21 


Poem 1 
Forgive my 
Guilt 








3.89* 


1.79 


-.76 


Poem 2 
Dandelions 










-1.08 


-4.07 


Non-fiction 
The Iks 


1 










-2.04 


Non-fiction 2 
The Birth of 
the Moon 



* P < .05 
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TABLE 2: SHORT ANSW ER MEASURES ( SCALE Q 3) 



ITEM DIFFICULTY 



TEXT / ITEMS 


1 


2 


3 


4 


5 


6 


7 


a 
a 


Story 1 
The Man by 
the Fountain 


2.06 


2.12 


2.23 


2.11 


2.16 


1.92 


2.08 


2.00 


Story 2 
The Use of 
Force 


1.85 


1.93 


2.21 


2.21 


2.20 


2.10 


1.95 


2.07 


Poem 1 
Forgive my 
Guilt 


2.13 


2.14 


1.88 


2.01 


2.05 


1.82 


1.90 


2.02 


Poem 2 
Dandelions 


1.74 


1.84 


2.00 


1.55 


1.64 


1.73 


1.67 


1.89 


Non-fiction 1 
The Iks 


1.80 


2.05 


2.09 


1.83 


1.78 


1.69 


1.73 


1.83 


Non-fiction 2 
The Birth of 
the Moon 


2.47 


2.02 


2.46 


1.99 


1.89 


1.93 


1.76 


2.30 
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TABLE 3a: MULTIPLE C HOICE MEASURES (SCALE 0 81 





Means 


Standard Deviation 


Story 1 
The Man by 
the Fountain 


4.15 


2.23 


story 2 
The Use of 
Force 


5.90 


1.97 


Poem 1 
Forgive my 
Guilt 


5.21 


1.73 


Poem 2 
Dandelions 


5.24 


1.4S 


Non-fiction 1 
The Iks 


5.64 


1.52 


Non-fiction 2 
The Birth of 
the Moon 


4.87 


1.66 
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TABLE 3b: T-TEST FOR MULTIPLE CHOICE ME ASURES ACROSS 6 TEXTS 

STORY POEM NON-FICTION 



1 


2 1 


2 


1 


2 


story 1 
The Man by 
the Fountain 


-2.58* -1.86 


-1.96 


-2.66* 


-1.21 


Story 2 
The Use of 
Force 


1.25 


1.26 


.48 


1.83 


Poem 1 
Forgive my 
Guilt 




-.06 


-.95 


.72 


Poem 2 
Dandelions 






-.94 


.82 


Non-fiction 1 
The Iks 








1.68 


Non-fiction 2 
The Birth of 
the Moon 



* P < .05 
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TABLE 4: MULTIPLE CH OICE MEASURES 



ITEM DIFFICULTY 



TEXT / ITEMS 


1 


2 


3 


4 


5 


6 


7 


8 


Story l 
The Man by 
the Fountain 


.60 


.40 


.50 


.45 


.75 


.50 


.45 


.50 


Story 2 
The Use of 
Force 


.79 


.84 


.74 


.84 


.68 


.37 


.84 


.79 


Poem 1 
Foraive mv 

GU.Ut 


.71 


.86 


1.00 


.64 


.64 


.43 


.32 


.61 


Poem 2 
Dandelions 


.88 


.88 


.44 


.76 


.56 


.44 


.48 


.80 


Non-fiction 1 
The Iks 


.88 


.80 


.84 


.64 


.44 


.60 


.60 


.84 


Non-fiction 2 
The Birth of 
the Moon 


.96 


.83 


.87 


.35 


.13 


.91 


.56 


.26 
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TABLE 5a: ESSAY 


(LITERATURE 


SCORES) (SCALE 0 4) 




Means 


Standard Deviation 


The Man by 
the Fountain 


1.73 


1.33 


Story « 

The Use of 
Force 


2.44 


1.54 


jroem x 
Forgive my 
Guilt 


2.56 


1.10 


jfoem * 
Dandelions 


1.89 


.87 


Non-fiction 1 
The Iks 


1.84 


1.38 


Non-fiction 2 
The Birth of 
the Moon 


2.39 


1.42 
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TABLE 5b: T-TEST FOR ESSAY ( LITERATURE SCORE) ACR OSS 6 TEXTS 



STORY POEM NON-FICTION 

1 2 12 12 



Story 1 
The Man by 
the Fountain 


-1.40 


-1.94 


-.42 


-.23 


-1.36 


Story 2 
The Use of 
Force 




-.25 


1.34 


1.25 


.11 


Poem 1 
Forgive my 
Guilt 






2.03* 


1.73 


.39 


Poem 2 
Dandelions 








.14 


-1.28 


Non-fiction 1 
The Iks 










-1.19 


Non-fiction 2 
The Birth of 
the Moon 



* P < .05 
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TABLE 6a: E S S AY 


t WRITING SCORES) 


( SCALE 0 --- 4) 




MEAN 


STANDARD DEVIATION 


Storv l 

maw \0 m mw mm J 

The Man by 
the Fountain 


1.73 


1.39 


Storv 2 
The Use of 
Force 


2.04 


1.61 


Poem 1 
Forgive my 
Guilt 


2.56 


.98 


Poem 2 

Dandelions 


2.15 


.96 


Non-fiction 1 
The Iks 


1.72 


1.17 


Non-fiction 2 
The Birth of 
the Moon 


1.92 


1.35 
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TABLE 6b: T-TEST 


FOR ESSAY 


(WRITING SCORE) ACROSS 6 TEXTS 


1 


STORY 

2 


POEM NON-FICTION 
12 12 


Story 1 
The Man by 
the Fountain 


-.61 


-1.99 -1.05 .03 -.41 


story 2 
The Use of 
Force 




-1.17* -.27 .80 .29 


Poem 1 
Forgive my 
Guilt 




1.25 2.46* 1.70 


Poem 2 
Dandelions 




1.32 .66 


Non-fiction 1 
The Iks 




-.55 


Non-fiction 2 
The Birth of 
the Moon 



* P < .05 
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TABLE 7: AESTHETIC JUDGEMENT SCORES FOR S TORY 1 AND 2 

(THE MAN BY THE FOUNTAIN / THE USE OF FORCE \ 



FACTOR* SCORES 





Means 


Standard Deviations 


Factor 1: 






Positive Aesthetic 


20.04 


6.36 


Rating 






(Scale 0 32) 






Factor 2: 
Negative Aesthetic 
Rating 

(Scale 0 28) 


13.8 


5.44 



TABLE 8: 

FACTOR LO ADINGS FO R AESTHETIC JUDGEMENT FOR STORY 1 AND 2 
(THE MAN BY THE FOUNTAIN / THE USE OF FORCE) 



Factor 1 Loadings** Factor 2 Loadings 

(Positive Rating) (Negative Rating) 



Teaching 


.79 


Awkward 


.59 


Logical 


.64 


Generic failure 


.62 


Symbolically 


.64 


Immoral 


.83 


Traditionally 
appropriate 


.76 


Unfulfilled 
intentions 


.81 


Original 


.84 


Unrealistic 


.74 


Richly 
ambiguous 


.82 


Irrelevant 


.73 


Imaginative 


.76 


Incongruent 
view 


.81 


Honest and true 


.82 






*. Only above 10% 
**. Only above .5 


of the variance are considered 
factor loadings are considered 
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TABLE 9: AESTHETIC JUDGEMENT SCORES FOR POEMS 1 AND 2 

( FORGIVE MY GUILT / DANDELIONS 1 



FACTOR* SCORES 





Means 


Standard Deviations 


Factor 1: 
Negative Aesthetic 
Rating 1 
(Scale 0 — 20) 


10.11 


1.92 


Factor 2: 
Negative Aesthetic 
Rating 2 
(Scale 0 12) 


6.86 


1.10 


Factor 3: 
Positive Aesthetic 
Rating 

(Scale 0 16) 



TABLE 10s 

FACTOR ' LOADINGS FO R AESTHETIC JUDGEMENT FOR POEMS 1 AND 2 

(FOR GIVE MY GUILD / DANDELIONS ) 



Factor 1 Loadings** Factor 2 Loadings Factor Loadings 
(Negative Rating) (Negative Rating) (Positive Rating) 



Unin- 
teresting 


.78 


Generic 
failure 


.72 


Teaching 


.61 


Unrea- 
listic 


.60 


Unfulfilled 
intentions 


.84 


Logical 


.68 


Irre- 
levant 


.65 


Honest 
and true 


-.64 


Awkward 


-.54 


Honest 
and true 


.50 






Symbo- 
lically 


.58 


Incon- 
gruent 
view 


.74 











*. Only above 10% of the variance are considered 
**. Only above .5 factor loadings are considered 
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TABLE 11: AESTHETIC JUDGEMENT SCORES FOR NON-FICTION 1 AND 2 

(THE IKS / THE BIRTH OF THE MOON! 



FACTOR* SCORES 





Means 


standard Deviations 


Factor 1: 






Negative Aesthetic 


16.97 


6.19 


Rating 






(Scale 0 32) 






Factor 2: 
Positive Aesthetic 
Rating 

(Scale 0 32) 


19.94 


6.04 



TABLE 12: FACTOR LOADINGS FOR AESTHETIC JUCSFMENT 

FOR NON-FICTION 1 AND 2 
(THE IKS / THE BIRTH OF THE MOON* 



Factor 1 Loadings** Factor 2 Loadings 

(Negative Rating) (Positive Rating) 



Uninteresting 


.78 


Teaching 


.69 


Awkward 


.80 


Logical 


.78 


Generic failure 


.83 


Symbolically 


.82 


Immoral 


.72 


Traditionally 
appropriate 


.82 


Unfulfilled 
Intentions 


.75 


Original 


.76 


Unrealistic 


.72 


Richly 
ambiguous 


.79 


Irrelevant 


.76 


Imaginative 


.85 


Incongruent 
view 


.83 


Honest and ture 


.69 



*• Only above 10% of the variance are considered 
**. Only above .5 factor loadings are considered 
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TABLE 13: AESTHETIC J UDGEMENT SCORES ACROSS THE SIX TESTS 



FACTOR* SCORES 





Means 


Standard Deviations 


Factor 1: 






Negative Aesthetic 


15.64 


5.41 


Rating 






(Scale 0 32) 






Factor 2: 
Positive Aesthetic 
Rating 

(Scale 0 32) 


20.71 


5.41 



TABLE 14: 

FACTOR LOADINGS FOR AESTHETIC JUDGEMENT 
ACROSS THE S IX TEXTS 

Factor 1 Loadings** Factor 2 Loadings 

(Negative Rating) (Positive Rating) 



uninteresting 


.66 


Teaching 


.71 


Awkward 


.71 


Logical 


.67 


Generic failure 


.72 


Symbolically 


.66 


Immoral 


.69 


Traditionally 
appropriate 


.69 


Unfulfilled 
intentions 


.73 


Original 


.77 


Unrealistic 


.77 


Richly 
ambiguous 


.74 


Irrelevant 


.74 


Imaginative 


.78 


Incongruent 
view 


.77 


Honest and ture 


.73 



*. Only above 10% of the variance are considered 
**. Only above .5 factor loadings are considered 
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TABLE 15: MEAN RATING OF QUALITY (SCALE 0 61 



Mean rating of quality 



Story l 
The Man oy 
the Fountain 


3 .50 


Story 2 
The Use of 
Force 


3 . 00 


Poem 1 
Forgive my 
Guilt 


4.86 


Poem 2 
Dandelions 


3.71 


Non-fiction 1 
The Iks 


3.06 


Non-fiction 2 
The Birth of 
the Moon 


3.50 
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TABLE 16: TEXT 


ATTITUDE MEASURES 


f SCALE 0 32 ) 




Means 


Standard Deviation 


Story 1 
The Man by 
the Fountain 


17.21 


9.59 


Story 2 
The Use of 
Force 


17.54 


8.34 


Poem 1 
Forgive my 
Guilt 


21.38 


2.73 


Poem 2 
Dandelions 


20.62 


4.97 


Non-fiction 1 
The Iks 


15.56 


8.13 


Non-fiction 2 
The Birth of 


15.50 


10.33 



the Moon 
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TABLE 17: BACKGROUND (MYTH! (0 12) 

Means Standard Deviation 

6.14 2.64 

ITEM DIFFICULTY 

12 3 4 567 89 10 11 12 
.31 .87 .32 .46 .81 .62 .78 .52 .48 .21 .82 .24 



TABLE 18: BACKGROUND (TERMS) (SCALE 0 36) 

Means Standard Deviation 

19.47 8.05 



ITEM DIFFICULTY ( 0 3 ) 

123456789 10 11 12 
1.53 2.02 1.08 1.98 1.64 2.03 1.91 1.83 0.85 1.02 1.62 1.95 
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TABLE 19: ATTITUDE TRANSFER AND INTEREST 

(SCALE 0 — 40) 



Transfer Interest 

Means standard Deviation Means Standard Deviation 
23.58 6.15 18.34 4.98 



ITEM DIFFICULTY (Transfer 0 4) 

123456789 10 

2.42 2.99 2.68 1.91 2.49 1.90 2.35 1.89 2.73 2.20 

ITEM DIFFICULTY (Interest 0 4) 

123456789 10 

2.63 1.42 2.58 1.43 2.04 1.98 2.30 2.61 1.82 2.12 
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TABLE 20: FACTOR ANALYSIS FOR CENSORSHIP 



FACTOR* SCORES 

Means Standard Deviations 

Factor 1: 

11.47 3.48 

(Scale 0 24) 



Factor 1 


Loadings** 


(Right to Shield the Young) 




Elementary protection 


.55 


Moral protection 


.76 


Taxpayers right 


.55 


Parent* s right 


.54 


Crime and destruction 


.74 


*. Only above 10% of the variance 
**. Only above .5 factor loadings 


are considered 
are considered 
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TABLE 21 COMPARISON OF LITERARY AND NON- LITERARY 

AESTHETIC JUDGEMENT ON SIX TEXTS 



• 


NUMBER OF ITEMS* 


SELECTED BY MOST 


STUDENTS 




Literary 


Non-Literary 


Mixed 


story 1 
The Man by 
the Fountain 


2 


1 




Story 2 
The Use of 
Force 


1 


2 


1 


Poem 1 
Forgive my 
Guilt 


3 


0 




Poem 2 
Dandelions 


2 


0 




Non-fiction 1 
The Iks 


2 


3 




Non-fiction 2 
The Birth of 
the Moon 




1 


1 



* There are 16 aesthetic judgement items. 
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BIVARIATE STATISTICS 
TABLE 22 

CORRELATION OF OPEN-E NDED AND MULTIPLE CHOICE QUESTIONS 

CONCERNING TEXTS 



Story Poem Non-fiction 



Story 


.10 
N - 39 






Poem 




.37* 
N - 53 




Non- 
fiction 






.53* 
N - 48 



* P < .05 
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6S 



CORRELATION OF SHORT AND EXTEND ED STUDENT RESPONSES 

CONCERNING TEXTS 



TABLE 23 





Story 


Poem 


Non-fiction 


Story 


.53* 
N - 33 






Poem 




.27 
N - 37 




Non- 
fiction 






.25 
N - 37 



* P < .05 



67 

70 



TABLE 24 



story 


Poem 


Non-Fiction 


story .89* 

N « 33 


Poem 


.72* 
N - 37 




Non-fiction 




.82* 
N - 37 



P < .05 
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TABLE 25 



COMPARISON OF KNOWLEDGE AND PRACTICE 



Background 



Myth 



Terms 



Story 1 
The Man by 
the Fountain 

Story 2 
The Use of 
Force 



N 



.25 

i 39 



N 



.18 

> 16 



Poem 1 
Forgive my 
Guilt 



.26* 
H ■ 51 



Poem 2 
Dandelions 



.49* 
N - 25 



Non-fiction 1 
The Iks 



N 



.29 

' 21 



Non-fiction 2 
The Birth of 
the Moon 



.69* 
N « 18 



P < .05 
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:RLC 
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TABLE 26 



CORRELATION OF PRACTICE AND PREFERENCE 





Positive 

aesthetic 

rating 


Negative 

aesthetic 

rating 


General 
rating 


Literary 
charac- 
teristic 


N 


Story 1 
The Man by 
tha Fountain 


. 08 


. 08 


/ 

. 04 


-.06 


26 


Story 2 
The Use of 
Force 


-.03 


. 56* 


. 47* 


• 32 


24 


Poem 1 
Forgive my 
Guilt 


.19 


-.26 


.39* 


.09 


24 


Poem 2 
Dandelions 


.17 


-.54* 


.15 


-.03 


21 


Non-fiction l 
The Iks 


.11 


-.02 


.15 


.27 


18 


Non-fiction 2 
The Birth of 
the Moon 


-.25 




-.26 


-.40* 


18 



* P < .05 
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TABLE 27 



COMPARISON OF PRACTICE WITH ATTITUDES 



Transfer Interest Censorship 

Story 1 .27* .22 

The Man by N «= 40 N ■ 40 

the Fountain 

Story 2 - 26 

The Use of N - 16 

Force 

Poem 1 -.05 .13 

Forgive my N ■ 52 N ■ 52 

Guilt 

Poem 2 -.02 
Dandelions N - 25 



Non-fiction 1 .05 .36 

The Iks N - 21 N ■ 21 



Non-fiction 2 -.41* 
The Birth of N - 18 

the Moon 



.05 
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I 3 

ERJC 
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TABLE 28 



CORRELATION OF SHORT ANSWER SCORES 
LITERAR Y AND NON-LITERARY TEXTS 





Story Poem 


Non-fiction 


Story 


.49* 


.48* 




N - 42 


N - 50 


Poem 




.49* 






N - 51 



* P < .05 
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